Building a Personal AI Newspaper

The Problem With News

I've grown increasingly frustrated with how I consume news. Not because there's too little information; quite the opposite. The internet drowns us in a firehose of content, most of which is irrelevant noise. Traditional news aggregators like Google News or Apple News try to solve this by learning what you click on, but their incentives aren't aligned with mine. They want engagement; I want to be informed about things that actually matter to me.

What I really wanted was a personal newspaper. Something that scans hundreds of RSS feeds, filters ruthlessly based on my actual interests (not just click patterns), and presents me with a clean, readable HTML page every morning. No infinite scroll. No algorithmic manipulation. Just 5–6 stories that genuinely match what I care about.

So I built one.

The Architecture (Simple in Theory)

The basic pipeline looks straightforward enough:

1. Collect articles from 30+ RSS feeds
2. Pre-filter them locally to find relevant candidates
3. Curate the final selection using an LLM
4. Deploy the result to a Raspberry Pi connected to an e-paper display on my wall

The RSS collection part was trivial: feedparser handles the heavy lifting, and I just needed to normalize the various feed formats into a common structure. I defined my interests in a YAML file with weighted categories:

Technology:
  - 8 AI+technology
  - 7 device+new and unusual
  - 6 Space+optimism

Science:
  - 8 Travel+parallel dimensions
  - 6 Antigravity

The weights (1–10) indicate how much I care about each topic. The + notation combines concepts: an article needs to be about both AI and technology, not just one. This is where things got interesting.

The Prefilter Problem

My first approach was to throw each headline at a small local LLM (Ollama's phi3:mini) and ask it to score relevance on a 1–10 scale. This worked, but it was slow (one API call per headline), non-deterministic, and the model kept hallucinating relevance where there was none.

The obvious alternative: embedding vectors. Embed all my interest topics once, embed each incoming headline, compute cosine similarity, done. This is fast, deterministic, and doesn't require an LLM for the filtering step.

Except it introduced a subtle but infuriating bias.

The Single-Concept Bias

Here's the problem I didn't anticipate: single-concept topics systematically outcompete multi-concept topics.

Consider these two interest specifications:

• Sweden (single concept)
• AI+technology (multi-concept, AND-combined)

For a headline to match "AI+technology", it needs to be semantically close to both "AI" and "technology" simultaneously. But for "Sweden", it just needs to be close to one thing. In embedding space, this creates an unfair advantage. The "Sweden" vector is a single point; the "AI+technology" requirement defines a much smaller region of space where both constraints are satisfied.

Result: Sweden-related articles kept winning against genuinely interesting AI stories, even though I weighted AI higher.

This is not a bug in the embedding model. It's a fundamental geometric property of how AND-constraints work in vector spaces.

The Calibration Experiments

I set up an offline experiment harness to diagnose this. The setup:

1. Cache a pool of ~1200 real RSS headlines
2. Select 10 diverse headlines using farthest-point sampling in embedding space
3. Test 9 different scoring mechanisms against all topics
4. Count how often single-concept vs multi-concept topics win

Here's what I tested:

The results were illuminating:

M1 (centroid cosine) almost completely fixed the bias: multi-concept topics started winning more often. But this came with a new problem: the centroid approach was too lenient. It would confidently match articles to topics even when the match was essentially random noise.

The No-Match Gate

I needed a way to detect "this doesn't actually match anything" versus "this genuinely matches this topic". The insight came from thinking about it statistically: what does a "random" match look like?

For any headline, I can compute its similarity to all my concept embeddings: this gives me a distribution of "how similar is this headline to random concepts". If my best topic match isn't significantly better than drawing random concepts from this distribution, then it's probably not a real match.

The implementation uses a contrastive lift calculation:

lift = topic_min_sim - E[min_k_random]

Where:

• topic_min_sim is the minimum similarity to any concept in the best-matching topic
• E[min_k_random] is the expected minimum if you randomly drew k concepts from the full pool

This expectation can be computed exactly (no Monte Carlo needed) using combinatorics. If lift <= 0, the match is no better than random, and the article gets dropped.

The result: a prefilter that's both fair to multi-concept topics AND capable of rejecting genuinely irrelevant articles.

The Final Assembly

With the prefilter producing a clean list of 80 candidates, the final curation step is handled by Gemini 3 Pro (via Poe's API). The prompt includes:

• My full interests list with weights
• The pre-scored candidates with their matched topics
• Instructions to pick exactly 6 stories and generate a self-contained HTML newspaper

The LLM does what LLMs do best: synthesize, summarize, and make editorial judgments that would be hard to codify. It can follow links to gather context, pick images, and write actual headlines instead of just regurgitating RSS titles.

The output is validated (must be well-formed HTML with proper charset), then pushed via SFTP to a Raspberry Pi. The Pi is connected to a large e-paper display hanging on my wall. Every morning at 7 AM, it refreshes with a fresh front page. No glowing screens, no notifications, just ink on paper.

What I Learned

Vector similarity is not intuitive. The geometric properties of embedding spaces lead to surprising behaviors that require careful calibration. AND-constraints don't work the way you'd expect.

Local LLMs are not always the answer. For tasks that require consistency and speed, embedding-based approaches can outperform small instruction-following models. But you need to understand what you're trading away.

Random baselines are underrated. Many "matches" in vector similarity are essentially noise. Comparing against what random looks like is a powerful sanity check.

Personal tools deserve the same rigor as production systems. This is just a weekend project for my own news consumption, but the vector calibration work was essential to making it actually useful.

The Code

The full pipeline is about 600 lines of Python across a handful of modules:

• src/collector/rss.py: RSS fetching and normalization
• src/filter/vectors.py: Embedding-based scoring with lift gating
• src/curation/prompt.py: Gemini prompt construction
• src/curation/poe_client.py: Poe API wrapper
• src/deploy/pi_push.py: SFTP deployment to Raspberry Pi

Configuration lives in a single config.toml:

[ollama]
base_url = "http://localhost:11434"
model = "bge-m3:latest"

[curation]
target_count = 6
max_to_send = 80

[deploy]
enabled = true
host = "192.168.1.33"
remote_path = "/tmp/daily_news.html"

The interests are defined in interests.yaml with the category/weight/concept structure shown earlier.

Looking Forward

There are obvious improvements to make:

• Feedback loop: Track which stories I actually read (click) and use that to adjust topic weights over time
• Image handling: Currently relying on Gemini to find images; could pre-fetch og:image tags from article pages
• Archive: Save daily outputs with timestamps for retrospective analysis
• Mobile: The Raspberry Pi serves plain HTML; could add responsive CSS or a simple PWA wrapper

But the core system works. Every morning, I get exactly what I asked for: a handful of genuinely interesting articles, selected by my criteria, presented without manipulation.

In a world of infinite content and engagement-optimized feeds, there's something deeply satisfying about a tool that simply does what you tell it to.